1. Introduction and Goals

This project analyses the LendingClub's loan dataset. The dataset includes loan application data, the information on granted loans and their status. In total {ADD SIZE} loans are included in the dataset.

The goal of this project is build several ML models which can be used to predict:

• Binary loan_status prediction model. We have used XGBoost and CatBoost to build a model which can be used to predict how likely an applicant is to default on a loan based on their information and the loan size, type and other specifics. This model can be used to decide whether the application should be approved or used by a third party investor to estimate risk. The goal of this model is to maximize the quality of probability prediction rather than the classification accuracy itself. In addition to estimating the default likelihood we'll also use this model to estimate:
  • Loan grades (A,B,C etc.) based on probability threshold, this does not directly attempt to predict the grades defined in the dataset but rather can be used as an alternative method to classify loan risk
  • Using this model we'll calculate the profitability and risk of a loan portfolio if only and provide an improved (more restrictive) method of evaluating loan applicants
• loan_grade prediction models, we'll evaluate and compare two different approaches
  • Multi classification model using XGBoost and CatBoost
  • Regression model that attempts to predict grade converted to a normalized numerical variable (predictions are rounded). While using a regression model for a classification problem is not ideal because it ignores the fact that spacing between different classes of ordinal features might differ (e.g. we don't know if the distance between grade A and B is the same as between B and C etc.) {TODO finish}

Data Selection:

To limit the extent of this project we have only included loans which were fully paid back or have been default on by the lender in our analysis based on the loan_status column

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

1.1 EDA

This is the exploratory analysis of the dataset and includes all the features shown in the (Features) notebook.

The EDA was done in parallel with model design process to minimize the influence analyzing manually analyzing the interaction between features might have had on our model design process.

Data selection

We have three target variables which we'll try to predict using in our following sections. They are shown here as:

• target_loan_status: whether the loan applicant has defaulted on their loan (0 or 1)
• pass_grade: the grade assigned to the loan by the "Lenders Club" (A,B,C,D etc.)
• pass_interest_rate: the interest rate set for the loan (converted to float)

['target__loan_status']

1.1.1 Correlation Matrix

Loan Status has strong correlation with:

• Applicant FICO (credit) score
• Term of the loan (longer loans are somewhat more to be defaulted on)
• While the correlation with the zip_code is almost 0.4 it is not significant (other variables like grade, sub_grade and interest rate are not included in our model because they are calculated based on estimated loan risk, we could still include them if our goal is to verify the quality of "Lenders Club" estimates we chose not to include them to build a "clean" alternative model using the same source data).

Loan Grade and Interest Rate have significant correlations with a much larger number of variables:

• Loan Term
• Loan amount and installment size
• FICO score
• Loan purpose
• Applicant income
• Income verification status etc.

This would indicate that the Lenders Club is trying to use a large number of variables to determine the grade and interest rate but that those variables are not necessarily as useful at predicting the likelihood of default.

Because the datatypes of features vary we had to use different methods to measure the strength and significance of each pair:

- Chi-Squared Test: Assesses independence between two categorical variables.  For bool-bool pairs due to categorical nature.

- Point Biserial Correlation: Measures correlation between a binary and a continuous variable. For bool-numerical pairs to account for mixed data types.

- Spearman's Rank Correlation: Assesses monotonic relationship between two continuous variables. Used for numerical-numerical pairs (for non-normally distributed data).

Since the Chi-Squared test outputs an unbound statistic/value which can't be directly compared to  pointbiserialr or Spearman Rank we have converted them to a  ` Cramér's V:` value which is normalized between 0 and 1. This was done to make the values in the matrix more uniform however we must note that Cramér's V and Spearman's correlation coefficients are fundamentally different statistics and generally can't be directly compared.

1.1.1 Relationships between individual variables

This part includes various charts showing the relationships between specific manually selected variables:

We can see that much fewer grade A and B loans were defaulted on while at C or below the likelihood of default becomes higher than average:

'Default Rate and Expected Return by Grade:'

	default_rate	mean_interest_rate	expected_return
pass__grade
A	0.07	7.09	6.60
B	0.13	10.65	9.24
C	0.22	14.00	10.87
D	0.31	17.79	12.23
E	0.38	21.16	13.03
F	0.50	25.12	12.46
G	0.57	27.74	11.98

The expected_return show the rate of return adjusted by the default rate. e.g. while almost 60% of all G grade loans are defaulted on you can still expect to make ~12% per year because the interest is very high. A grade loans tend to be relatively much less profitable despite the much lower risk.

Text(0.5, 1.025, 'Grade and Interest Rate')

There is quite a bit of overlap between interest rates for lower grades (D+).

Text(0.5, 1.025, 'Grade and Loan Amount')

Text(0.5, 1.025, 'Grade and Annual Income')

As we would expect the applications by individuals who tend to have very low incomes tend to receive lower grades. However, the rate of E grade loans increases significantly along with grade A > ~100k yearly income which is quite interesting and requires additional analysis because this does not apply to loans of other grades (we can see in charts below that the income of ~25% applicants is > 90k so this can't really be explained by random variation due to small sample size)

Text(0.5, 1.025, 'Grade and Fico Score')

Text(0.5, 1.025, 'Default Rate and Interest Rate')

Text(0.5, 1.025, 'Default Rate and Annual Income')

While low income individuals are much more likely to default the risk of default doesen't seems to decrease much or at as income goes up.

Text(0.5, 1.025, 'Loan Amount and Purpose')

term_parser__term: int64
target__loan_status: float64
pass__loan_amnt: float64
pass__int_rate: float64
pass__grade: object
pass__home_ownership: object
pass__annual_inc: float64
pass__purpose: object
pass__fico_range_high: float64
pass__last_fico_range_high: float64

	term_parser__term
1233152	60
461729	36
738295	60
390716	36
350813	60
...	...
701647	36
1093222	36
1231982	36
143267	36
208225	36

5361 rows × 1 columns

term_parser__term_binned             category
term_parser__term                       int64
target__loan_status_binned           category
target__loan_status                   float64
pass__loan_amnt_binned               category
pass__loan_amnt                       float64
pass__int_rate_binned                category
pass__int_rate                        float64
pass__annual_inc_binned              category
pass__annual_inc                      float64
pass__fico_range_high_binned         category
pass__fico_range_high                 float64
pass__last_fico_range_high_binned    category
pass__last_fico_range_high            float64
pass__grade                            object
pass__home_ownership                   object
pass__purpose                          object
dtype: object

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

None

'term_parser__term_binned vs pass__fico_range_high No significant difference found (p-value = 0.302)'

None

None

None

None

None

None

None

None

1233152    714.0
461729     689.0
738295     694.0
390716     704.0
350813     694.0
           ...  
701647     719.0
1093222    749.0
1231982    719.0
143267     759.0
208225     664.0
Name: pass__fico_range_high, Length: 5361, dtype: float64

1.2.1 Variable Selection

1.2.2 Population Stability Index

We can use PSI to drop unstable features:

Interpreting PSI Values PSI = 0: No change in the distribution between the two datasets. 0 < PSI < 0.1: Indicates a minor change in distribution. Generally acceptable for most applications. 0.1 ≤ PSI < 0.25: Suggests a medium change, warranting investigation. PSI ≥ 0.25: Represents a significant change in the feature distribution, suggesting that the feature's behavior has altered considerably from the baseline. Features with high PSI are often considered unstable and may be dropped from the model to maintain its performance.

5361

DropHighPSIFeatures(split_frac=0.6)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

{'term_parser__term': 0.0,
 'emp_length_parser__emp_length': 0.00741761565898279,
 'target__loan_status': 0.0,
 'pass__loan_amnt': 0.025024201814442776,
 'pass__int_rate': 0.07176711417547209,
 'pass__installment': 0.011609246883005436,
 'pass__annual_inc': 0.03241740868044229,
 'pass__dti': 0.015406464288949512,
 'pass__delinq_2yrs': 0.0057098047008174195,
 'pass__fico_range_low': 0.024401601815981836,
 'pass__fico_range_high': 0.024401601815981836,
 'pass__inq_last_6mths': 0.0004806433251511103,
 'pass__open_acc': 0.020049125901470677,
 'pass__pub_rec': 0.0011239749466162331,
 'pass__revol_bal': 0.012375403776342789,
 'pass__revol_util': 0.010228320444652932,
 'pass__total_acc': 0.012150588779396194,
 'pass__last_fico_range_high': 0.018082420124089962,
 'pass__last_fico_range_low': 0.018082420124089962,
 'pass__collections_12_mths_ex_med': 0.0,
 'pass__inq_fi': 0.013248028575812698,
 'pass__inq_last_12m': 0.024401074697258687,
 'pass__chargeoff_within_12_mths': 0.0,
 'pass__mort_acc': 0.009757189838540613,
 'pass__pub_rec_bankruptcies': 4.868286594901147e-05,
 'pass__tax_liens': 0.0,
 'pass__tot_hi_cred_lim': 0.01626241771088676,
 'pass__total_bal_ex_mort': 0.020408531871268523}

2. Model Design

As we've discussed at the begining our goal is to build ML models to predict these target features:

• Loan Default Risk (loan_status)
• Loan Grade (loan_grade)
• Interest Rate (int_rate)

In all cases we've either used XGBoost or CatBoost models (based on performance and training time).

2.1. Model Pipelines

We've used sci-kit lear pipelines to build and design our models. The data passed to the pipeline is preprocessed separately in src/preproc which filters only completed loans (based on loan_status) and performs other clean up and preprocessing (without fundamentally transforming or scaling etc.) to make all the features compatible with our ML models.

Example of a pipeline:

Pipeline(steps=[('preprocessing',
                 FunctionTransformer(func=<function convert_to_category at 0x7f9a9fcefeb0>)),
                ('feat_trans_loan_grade', LoanGradeTransformer()),
                ('feat_trans_dti_inc_joint', JointApplicationTransformer()),
                ('feat_trans_loan_grade_risk', LoadGradeRiskTransformer()),
                ('feat_trans_fico_score', FICOScoreTransformer()),
                ('feat_trans_delinquency', Del...
                ('feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH',
                 DummyDropAllButFICOHigh()),
                ('feat_trans_dummytransformer',
                 DummyTestTransformer(option=<Options.OFF: 0>)),
                ('remove_columns',
                 FunctionTransformer(func=<function get_pipeline.<locals>.remove_columns_with_prefix at 0x7f9b2668d990>,
                                     kw_args={'prefix': 'target__'})),
                ('model',
                 <catboost.core.CatBoostClassifier object at 0x7f9a9bb53e80>)])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Each pipeline consists of these steps:

• a preprocessing function which handles data encoding if necessary (we do not encode categorical variables because XGBoost and CatBoost support them directly).
• a set of feature transformers selected based on hyper tuning results (the transformers were designed iteratively while designing the model and performing the EDA). The transformer merge/combine or otherwise modify various features based on subject knowledge (e.g. calculating various ratios or merging variables which represent basically the same features etc.)
• ML model.

2.2 Hyper-tuning

Because the dataset is very large we have split the tuning and model evaluation into separate notebooks, so they can be performed independently:

• Hyper tuning is first performed using default or fixed model hyperparameters to select the optimal transformers and their parameters.
• Hyperparameter tuning is performed using a fixed pipeline with transformers selected in the previous step.

This is a compromise approach, ideally we would use two nested loops with grid search performed on the outer level to select the transformers and a nested loop (using random or bayesian search) for model hyper tuning. However, considering the size of the dataset (and even if we use a relatively large sample) would be very slow because it would exponentially increase the number of required iterations.

The results are serialized to a binary file that includes the information on the model and training dataset.

2.3 Model Evaluation

Is performed by loading the hyper tuning information saved in the previous step, the model (with the same pipeline) is trained using the test training sample of the dataset, validated using CV and then evaluating using a separate training sample.

2.4 Production Deployment

The model from the previous step is serialized using (dill or joblib) and deployed to remote VM running a FastAPI backend that can be integrated into a website or other application.

2.1. Tuning the Binary Default Risk Classifier

We'll tune multiple XGBoost model which predict the loan_status column using F1 and PR-AUC (we have tried other metrics but they gennerally provide the same or inferior results)

Dropping transformer for ['term'], not in df
Total samples loaded for tuning: 1000

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

emp_length_parser__emp_length     float64
target__Accepted                    int64
zip__zip_code                    category
pass__loan_amnt                   float64
pass__addr_state                 category
pass__last_fico_range_high        float64
pass__dti                         float64
dtype: object

Tunning: - transformers: transformers: 1
 total options: 2
[<class 'shared.ml_config_core.DummyTestTransformer'>]
search_grid:
{'feat_trans_dummytransformer__option': [0, 1]}

 - model: XGBoostTuneF1Rejected n_iters=100 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4], 'model__scale_pos_weight': [1, 2, 3, 5, 10]}
Fitting 5 folds for each of 2 candidates, totalling 10 fits
Best Transformer Parameters: {'feat_trans_dummytransformer__option': 0}
Fitting 5 folds for each of 100 candidates, totalling 500 fits
Best Model Parameters: {'model__scale_pos_weight': 2, 'model__n_estimators': 200, 'model__min_child_weight': 3, 'model__max_depth': 4, 'model__learning_rate': 0.05, 'model__gamma': 0.4}
XGBoostTuneF1Rejected Fit, total time:11.0

Tunning: - transformers: transformers: 1
 total options: 2
[<class 'shared.ml_config_core.DummyTestTransformer'>]
search_grid:
{'feat_trans_dummytransformer__option': [0, 1]}

 - model: XGBoostTunePRAUCRejected n_iters=100 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4], 'model__scale_pos_weight': [1, 2, 3, 5, 10]}
Fitting 5 folds for each of 2 candidates, totalling 10 fits
Best Transformer Parameters: {'feat_trans_dummytransformer__option': 0}
Fitting 5 folds for each of 100 candidates, totalling 500 fits
Best Model Parameters: {'model__scale_pos_weight': 1, 'model__n_estimators': 200, 'model__min_child_weight': 1.5, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0}
XGBoostTunePRAUCRejected Fit, total time:9.4

	best_score	best_params
model_key
XGBoostTuneF1Rejected	0.919000	{'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 2, 'model__n_estimators': 200, 'model__min_child_weight': 3, 'model__max_depth': 4, 'model__learning_rate': 0.05, 'model__gamma': 0.4}
XGBoostTunePRAUCRejected	0.949714	{'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 1, 'model__n_estimators': 200, 'model__min_child_weight': 1.5, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0}

While we have tried LightGBM, CatBoost and other models initially (see shared/ml_config_core for various tested configurations) we've narrowed to these two configurations because they provided the best (or approximately the same as) results while also being the fastest to train.

The parameters tuned to maximize F1 generally provide the best performance for binary classification, however the PR-AUC configuration results in more accurate probabilities because (which might be preferable because our goal is to build a model to select the loans to invest in)

Preprocessing and Transformer Tuning

We have attempted to performance various feature transformations* to our dataset which could potentially improve performance (i.e. binning or merging certain variables etc.). Because we have a high number of different combinations and because invidual transformers might perform differently when tuning for certain metrics we've also included them in our tuning process included:

1. Grid search is performed with every combination of transformers using "semi-optimal" (based on previous tuning) hyperparameter defined for each specific model type.
2. Random search is used to select the optimal hyperparameter

• see src/feature_engineering/py for full list of transformers.

Results based on transformer combinations

0 = disabled, some transformers have multiple different parameters (e.g. 1,2..)

'XGBoostTuneF1Rejected:'

	feat_trans_dummytransformer__option	mean_test_score
0	0	0.911460
1	1	0.755133

'XGBoostTunePRAUCRejected:'

	feat_trans_dummytransformer__option	mean_test_score
0	0	0.941563
1	1	0.617895

We have built a simple regression model to evaluate the impact of each transformer and the hyperparameter. This is just to give a rough estimate and to allow comparing individual parameters (the values themselves are not particularly meaningful)

'Transformer impact for XGBoostTuneF1Rejected'

	Transformer_Option	Impact
0	feat_trans_dummytransformer__option_1	-0.1563

'Transformer impact for XGBoostTuneF1Rejected'

	Transformer_Option	Impact
26	model__learning_rate_0.5	0.0670
21	model__learning_rate_0.05	0.0649
23	model__learning_rate_0.2	0.0646
24	model__learning_rate_0.3	0.0631
28	model__learning_rate_0.7	0.0626
22	model__learning_rate_0.1	0.0614
27	model__learning_rate_0.6	0.0611
29	model__learning_rate_0.9	0.0586
25	model__learning_rate_0.4	0.0580
30	model__learning_rate_1	0.0579

'Transformer impact for XGBoostTunePRAUCRejected'

	Transformer_Option	Impact
0	feat_trans_dummytransformer__option_1	-0.3237

'Transformer impact for XGBoostTunePRAUCRejected'

	Transformer_Option	Impact
26	model__learning_rate_0.5	0.0332
28	model__learning_rate_0.7	0.0296
24	model__learning_rate_0.3	0.0287
22	model__learning_rate_0.1	0.0283
29	model__learning_rate_0.9	0.0273
23	model__learning_rate_0.2	0.0267
25	model__learning_rate_0.4	0.0266
27	model__learning_rate_0.6	0.0252
21	model__learning_rate_0.05	0.0244
30	model__learning_rate_1	0.0212

Parameter hypertuning results:

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
32	0.165019	0.041838	0.006870	0.000338	3	100	1.50	12	0.2	0.05	0.952121	0.977085	0.937812	0.953181	0.922726	0.948585	0.018059	1
28	0.194744	0.023847	0.007625	0.000556	5	200	1.00	5	0.1	0.10	0.955119	0.969295	0.936617	0.947440	0.929790	0.947652	0.013893	2
19	0.045076	0.003802	0.006470	0.000455	3	50	3.00	5	0.5	0.00	0.958089	0.968593	0.935548	0.958187	0.916980	0.947479	0.018685	3
88	0.183628	0.004996	0.007775	0.000197	5	250	1.50	5	0.2	0.00	0.953320	0.972358	0.933539	0.951259	0.926392	0.947374	0.016164	4
54	0.053619	0.003327	0.006009	0.000066	5	150	2.50	4	0.6	0.30	0.942307	0.976027	0.938191	0.955390	0.924076	0.947198	0.017538	5
61	0.105711	0.011028	0.006828	0.000353	2	150	2.00	4	0.2	0.05	0.955595	0.970291	0.942990	0.947695	0.917151	0.946744	0.017456	6
51	0.111597	0.004462	0.007071	0.000820	5	200	2.00	6	0.3	0.05	0.946948	0.974495	0.945376	0.954955	0.909044	0.946164	0.021256	7
17	0.145733	0.009177	0.006647	0.000188	10	200	1.50	6	0.1	0.30	0.947011	0.970784	0.929056	0.952683	0.931175	0.946142	0.015276	8
34	0.041295	0.001735	0.006348	0.000309	5	50	0.75	6	0.9	0.10	0.949947	0.967954	0.946001	0.961007	0.902168	0.945415	0.022988	9
29	0.142682	0.004814	0.007668	0.001212	10	100	1.50	10	0.2	0.05	0.944369	0.976487	0.940033	0.945203	0.919718	0.945162	0.018197	10

Because the hypertuning process is very slow (due to a very high number of fits required for transformer tuning and the size of the dataset) the tuning results are serialized in a binary file and used to train the test models separately.

2.1.1 Training and Testing the Default Risk Model

The dataset is pre-split into a 20% test sample and a 80% training sample (10% of which is used for tuning):

Dropping transformer for ['term'], not in df
Total Samples Loaded 10000

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

Training: XGBoostTuneF1 with: {'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 3, 'model__n_estimators': 100, 'model__min_child_weight': 0.5, 'model__max_depth': 7, 'model__learning_rate': 0.2, 'model__gamma': 0.3}
XGBoostTuneF1Rejected: 1.4 seconds
Training: XGBoostTunePRAUC with: {'feat_trans_dummytransformer__option': 0, 'model__scale_pos_weight': 3, 'model__n_estimators': 100, 'model__min_child_weight': 1.5, 'model__max_depth': 12, 'model__learning_rate': 0.2, 'model__gamma': 0.05}
XGBoostTunePRAUCRejected: 2.1 seconds

5-Fold CV validation results (metrics calculated using the training dataset):

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:995: UserWarning: The figure layout has changed to tight
  plt.tight_layout()

Results using the test sample split:

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	emp_length_parser__emp_length	0.670845
1	zip__zip_code	0.138800
2	pass__loan_amnt	0.034620
3	pass__addr_state	0.033216
4	pass__last_fico_range_high	0.072705
5	pass__dti	0.049815

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	emp_length_parser__emp_length	0.619929
1	zip__zip_code	0.127074
2	pass__loan_amnt	0.046823
3	pass__addr_state	0.051340
4	pass__last_fico_range_high	0.092524
5	pass__dti	0.062311

Pipeline(steps=[('preprocessing',
                 FunctionTransformer(func=<function convert_to_category at 0x7f5dcc4f3ac0>)),
                ('feat_trans_dummytransformer',
                 DummyTestTransformer(option=<Options.OFF: 0>)),
                ('remove_columns',
                 FunctionTransformer(func=<function get_pipeline.<locals>.remove_columns_with_prefix at 0x7f5db2d89cf0>,
                                     kw_args={'prefix': 'target__'})),
                ('model',
                 XGBClass...
                               feature_types=None, gamma=0.05, grow_policy=None,
                               importance_type=None,
                               interaction_constraints=None, learning_rate=0.2,
                               max_bin=None, max_cat_threshold=None,
                               max_cat_to_onehot=None, max_delta_step=None,
                               max_depth=12, max_leaves=None,
                               min_child_weight=1.5, missing=nan,
                               monotone_constraints=None, multi_strategy=None,
                               n_estimators=100, n_jobs=None,
                               num_parallel_tree=None, random_state=None, ...))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

	emp_length_parser__emp_length	zip__zip_code	pass__loan_amnt	pass__addr_state	pass__last_fico_range_high	pass__dti
977081	11.0	307	12000.0	GA	689.0	0.1570
2514607	3.0	037	1500.0	NH	668.0	0.0163
1511719	4.0	191	15000.0	PA	724.0	0.1405
2612676	0.0	074	15000.0	NJ	674.0	0.1626
638147	1.0	720	20000.0	AR	774.0	0.3885
...	...	...	...	...	...	...
3078441	0.0	271	1000.0	NC	563.0	1.5606
365833	9.0	380	35000.0	TN	579.0	0.2330
1211175	11.0	531	10000.0	WI	704.0	0.2801
628962	1.0	272	8000.0	NC	554.0	0.1181
3208689	0.0	027	1500.0	MA	0.0	0.0000

2000 rows × 6 columns

2.1. Tuning the Binary Default Risk Classifier

We'll tune multiple XGBoost model which predict the loan_status column using F1 and PR-AUC (we have tried other metrics but they gennerally provide the same or inferior results)

Sampled n: 33910
Total samples loaded for tuning: 20301

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

Tunning: - transformers: transformers: 0
 total options: 0
[]
search_grid:
{}

 - model: XGBoostF1Base n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4], 'model__scale_pos_weight': [1, 2, 3, 5, 10]}
Fitting 5 folds for each of 70 candidates, totalling 350 fits
Best Model Parameters: {'model__scale_pos_weight': 3, 'model__n_estimators': 50, 'model__min_child_weight': 2, 'model__max_depth': 10, 'model__learning_rate': 0.05, 'model__gamma': 0.05}
XGBoostF1Base Fit, total time:123.7

Tunning: - transformers: transformers: 3
 total options: 6
[<class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.DropLowImportanceFeatures'>, <class 'src.feature_engineering.LoanPurposeRiskTransformer'>]
search_grid:
{'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_DropLowImportanceFeatures__option': [0, 1], 'feat_trans_loan_purpose__option': [0, 1]}

 - model: XGBoostTuneF1 n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4], 'model__scale_pos_weight': [1, 2, 3, 5, 10]}
Fitting 5 folds for each of 8 candidates, totalling 40 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/externals/loky/process_executor.py:752: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  warnings.warn(
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
20 fits failed out of a total of 40.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:40] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f3978d8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f3978fe0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f3978ff3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f3978fe4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f3978fe70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f3978fe91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f3978f995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f3978d7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f399d57ca4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:41] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fa8a0b8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fa8a0de0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fa8a0df3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fa8a0de4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fa8a0de70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fa8a0de91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fa8a0d995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fa8a0b7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fa8c5388a4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:42] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f6e6a78c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f6e6a9e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f6e6a9f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f6e6a9e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f6e6a9e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f6e6a9e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f6e6a9995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f6e6a77fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f6e8ef97a4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:42] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fb570d8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fb570fe0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fb570ff3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fb570fe4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fb570fe70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fb570fe91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fb570f995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fb570d7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fb5955e7a4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:43] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f977e38c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f977e5e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f977e5f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f977e5e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f977e5e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f977e5e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f977e5995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f977e37fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f97a01b5a4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:43] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fc0f0d8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fc0f0fe0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fc0f0ff3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fc0f0fe4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fc0f0fe70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fc0f0fe91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fc0f0f995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fc0f0d7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fc100d50a4a]



--------------------------------------------------------------------------------
2 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:41] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fb5c6f8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fb5c71e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fb5c71f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fb5c71e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fb5c71e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fb5c71e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fb5c71995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fb5c6f7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fb5d95dba4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:41] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f58fcf8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f58fd1e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f58fd1f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f58fd1e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f58fd1e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f58fd1e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f58fd1995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f58fcf7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f590bb41a4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:41] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f3978d8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f3978fe0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f3978ff3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f3978fe4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f3978fe70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f3978fe91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f3978f995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f3978d7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f399d57ca4a]



--------------------------------------------------------------------------------
4 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:45] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fb5c6f8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fb5c71e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fb5c71f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fb5c71e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fb5c71e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fb5c71e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fb5c71995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fb5c6f7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fb5d95dba4a]



--------------------------------------------------------------------------------
5 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:45] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fa8a0b8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fa8a0de0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fa8a0df3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fa8a0de4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fa8a0de70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fa8a0de91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fa8a0d995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fa8a0b7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fa8c5388a4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:25:45] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f6e6a78c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f6e6a9e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f6e6a9f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f6e6a9e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f6e6a9e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f6e6a9e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f6e6a9995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f6e6a77fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f6e8ef97a4a]



  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [0.76538169 0.76549475        nan        nan 0.76538169 0.76549475
        nan        nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1}
Fitting 5 folds for each of 70 candidates, totalling 350 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/externals/loky/process_executor.py:752: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  warnings.warn(

Best Model Parameters: {'model__scale_pos_weight': 2, 'model__n_estimators': 200, 'model__min_child_weight': 2, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostTuneF1 Fit, total time:160.0

Tunning: - transformers: transformers: 1
 total options: 1
[<class 'src.feature_engineering.DummyDropFICO'>]
search_grid:
{'feat_trans_dummy_drop_FICO__option': [1]}

 - model: XGBoostTuneF1DummyDropFICO n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4], 'model__scale_pos_weight': [1, 2, 3, 5, 10]}
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Best Transformer Parameters: {'feat_trans_dummy_drop_FICO__option': 1}
Fitting 5 folds for each of 70 candidates, totalling 350 fits
Best Model Parameters: {'model__scale_pos_weight': 5, 'model__n_estimators': 100, 'model__min_child_weight': 2, 'model__max_depth': 5, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostTuneF1DummyDropFICO Fit, total time:141.4

Tunning: - transformers: transformers: 0
 total options: 0
[]
search_grid:
{}

 - model: XGBoostPRAUCBase n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4], 'model__scale_pos_weight': [1, 2, 3, 5, 10]}
Fitting 5 folds for each of 70 candidates, totalling 350 fits
Best Model Parameters: {'model__scale_pos_weight': 10, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 12, 'model__learning_rate': 0.3, 'model__gamma': 0.3}
XGBoostPRAUCBase Fit, total time:110.7

Tunning: - transformers: transformers: 3
 total options: 6
[<class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.DropLowImportanceFeatures'>, <class 'src.feature_engineering.LoanPurposeRiskTransformer'>]
search_grid:
{'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_DropLowImportanceFeatures__option': [0, 1], 'feat_trans_loan_purpose__option': [0, 1]}

 - model: XGBoostTunePRAUC n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4], 'model__scale_pos_weight': [1, 2, 3, 5, 10]}
Fitting 5 folds for each of 8 candidates, totalling 40 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
20 fits failed out of a total of 40.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
3 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:28] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f3978d8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f3978fe0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f3978ff3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f3978fe4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f3978fe70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f3978fe91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f3978f995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f3978d7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f399d57ca4a]



--------------------------------------------------------------------------------
2 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:28] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fb570d8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fb570fe0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fb570ff3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fb570fe4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fb570fe70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fb570fe91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fb570f995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fb570d7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fb5955e7a4a]



--------------------------------------------------------------------------------
2 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:28] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fc0f0d8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fc0f0fe0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fc0f0ff3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fc0f0fe4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fc0f0fe70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fc0f0fe91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fc0f0f995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fc0f0d7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fc100d50a4a]



--------------------------------------------------------------------------------
2 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:28] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f977e38c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f977e5e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f977e5f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f977e5e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f977e5e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f977e5e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f977e5995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f977e37fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f97a01b5a4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:28] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fa8a0b8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fa8a0de0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fa8a0df3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fa8a0de4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fa8a0de70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fa8a0de91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fa8a0d995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fa8a0b7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fa8c5388a4a]



--------------------------------------------------------------------------------
4 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:32] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f977e38c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f977e5e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f977e5f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f977e5e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f977e5e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f977e5e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f977e5995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f977e37fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f97a01b5a4a]



--------------------------------------------------------------------------------
3 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:32] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f3978d8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f3978fe0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f3978ff3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f3978fe4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f3978fe70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f3978fe91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f3978f995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f3978d7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f399d57ca4a]



--------------------------------------------------------------------------------
2 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:32] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7f6e6a78c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7f6e6a9e0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7f6e6a9f3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7f6e6a9e4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7f6e6a9e70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7f6e6a9e91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7f6e6a9995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7f6e6a77fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7f6e8ef97a4a]



--------------------------------------------------------------------------------
1 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 427, in fit
    self._final_estimator.fit(Xt, y, **fit_params_last_step)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 1500, in fit
    train_dmatrix, evals = _wrap_evaluation_matrices(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 521, in _wrap_evaluation_matrices
    train_dmatrix = create_dmatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/sklearn.py", line 958, in _create_dmatrix
    return QuantileDMatrix(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1529, in __init__
    self._init(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 1590, in _init
    _check_call(ret)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [19:32:32] /home/conda/feedstock_root/build_artifacts/xgboost-split_1703076519788/work/src/data/../common/../data/gradient_index.h:94: Check failed: valid: Input data contains `inf` or a value too large, while `missing` is not set to `inf`
Stack trace:
  [bt] (0) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x6e) [0x7fa8a0b8c2ee]
  [bt] (1) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e0e44) [0x7fa8a0de0e44]
  [bt] (2) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(void xgboost::GHistIndexMatrix::PushAdapterBatch<xgboost::data::ArrayAdapterBatch>(xgboost::Context const*, unsigned long, unsigned long, xgboost::data::ArrayAdapterBatch const&, float, xgboost::common::Span<xgboost::FeatureType const, 18446744073709551615ul>, double, unsigned long)+0x705) [0x7fa8a0df3c05]
  [bt] (3) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(+0x3e4d72) [0x7fa8a0de4d72]
  [bt] (4) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::InitFromCPU(xgboost::Context const*, xgboost::BatchParam const&, void*, float, std::shared_ptr<xgboost::DMatrix>)+0x2195) [0x7fa8a0de70c5]
  [bt] (5) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::data::IterativeDMatrix::IterativeDMatrix(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x73b) [0x7fa8a0de91fb]
  [bt] (6) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create<void*, void*, void (void*), int (void*)>(void*, void*, std::shared_ptr<xgboost::DMatrix>, void (*)(void*), int (*)(void*), float, int, int)+0x7e) [0x7fa8a0d995be]
  [bt] (7) /home/paulius/miniconda3/envs/rapids_v2/lib/libxgboost.so(XGQuantileDMatrixCreateFromCallback+0x1b7) [0x7fa8a0b7fab7]
  [bt] (8) /home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/lib-dynload/../../libffi.so.8(+0x6a4a) [0x7fa8c5388a4a]



  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [0.81951084 0.82086583        nan        nan 0.81951084 0.82086583
        nan        nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1}
Fitting 5 folds for each of 70 candidates, totalling 350 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/externals/loky/process_executor.py:752: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  warnings.warn(

Best Model Parameters: {'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0.05}
XGBoostTunePRAUC Fit, total time:139.8

	best_score	best_params
model_key
XGBoostF1Base	0.769397	{'model__scale_pos_weight': 3, 'model__n_estimators': 50, 'model__min_child_weight': 2, 'model__max_depth': 10, 'model__learning_rate': 0.05, 'model__gamma': 0.05}
XGBoostTuneF1	0.773237	{'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 200, 'model__min_child_weight': 2, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostTuneF1DummyDropFICO	0.397436	{'feat_trans_dummy_drop_FICO__option': 1, 'model__scale_pos_weight': 5, 'model__n_estimators': 100, 'model__min_child_weight': 2, 'model__max_depth': 5, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostPRAUCBase	0.765507	{'model__scale_pos_weight': 10, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 12, 'model__learning_rate': 0.3, 'model__gamma': 0.3}
XGBoostTunePRAUC	0.829786	{'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0.05}

While we have tried LightGBM, CatBoost and other models initially (see shared/ml_config_core for various tested configurations) we've narrowed to these two configurations because they provided the best (or approximately the same as) results while also being the fastest to train.

The parameters tuned to maximize F1 generally provide the best performance for binary classification, however the PR-AUC configuration results in more accurate probabilities because (which might be preferable because our goal is to build a model to select the loans to invest in)

Preprocessing and Transformer Tuning

We have attempted to performance various feature transformations* to our dataset which could potentially improve performance (i.e. binning or merging certain variables etc.). Because we have a high number of different combinations and because invidual transformers might perform differently when tuning for certain metrics we've also included them in our tuning process included:

1. Grid search is performed with every combination of transformers using "semi-optimal" (based on previous tuning) hyperparameter defined for each specific model type.
2. Random search is used to select the optimal hyperparameter

• see src/feature_engineering/py for full list of transformers.

Results based on transformer combinations

0 = disabled, some transformers have multiple different parameters (e.g. 1,2..)

'XGBoostTuneF1:'

	feat_trans_DropLowImportanceFeatures__option	feat_trans_inst_income_ratio__option	feat_trans_loan_purpose__option	mean_test_score
1	0	0	1	0.765495
5	1	0	1	0.765495
0	0	0	0	0.765382
4	1	0	0	0.765382
2	0	1	0	NaN
3	0	1	1	NaN
6	1	1	0	NaN
7	1	1	1	NaN

'XGBoostTuneF1DummyDropFICO:'

	feat_trans_dummy_drop_FICO__option	mean_test_score
0	1	0.381916

'XGBoostTunePRAUC:'

	feat_trans_DropLowImportanceFeatures__option	feat_trans_inst_income_ratio__option	feat_trans_loan_purpose__option	mean_test_score
1	0	0	1	0.820866
5	1	0	1	0.820866
0	0	0	0	0.819511
4	1	0	0	0.819511
2	0	1	0	NaN
3	0	1	1	NaN
6	1	1	0	NaN
7	1	1	1	NaN

We have built a simple regression model to evaluate the impact of each transformer and the hyperparameter. This is just to give a rough estimate and to allow comparing individual parameters (the values themselves are not particularly meaningful)

'Transformer impact for XGBoostTuneF1'

	Transformer_Option	Impact
2	feat_trans_loan_purpose__option_1	0.0018
0	feat_trans_DropLowImportanceFeatures__option_1	0.0000
1	feat_trans_inst_income_ratio__option_1	-0.0058

'Transformer impact for XGBoostTuneF1'

	Transformer_Option	Impact
26	model__learning_rate_0.5	0.1620
21	model__learning_rate_0.05	0.0980
25	model__learning_rate_0.4	0.0969
4	model__n_estimators_100	0.0891
5	model__n_estimators_150	0.0868
24	model__learning_rate_0.3	0.0867
30	model__learning_rate_1	0.0844
22	model__learning_rate_0.1	0.0733
28	model__learning_rate_0.7	0.0712
10	model__min_child_weight_0.75	0.0687

Failed: XGBoostTuneF1DummyDropFICO

'Transformer impact for XGBoostTunePRAUC'

	Transformer_Option	Impact
2	feat_trans_loan_purpose__option_1	0.0059
1	feat_trans_inst_income_ratio__option_1	0.0039
0	feat_trans_DropLowImportanceFeatures__option_1	0.0000

'Transformer impact for XGBoostTunePRAUC'

	Transformer_Option	Impact
14	model__min_child_weight_2.5	0.0475
11	model__min_child_weight_1	0.0473
15	model__min_child_weight_3	0.0435
12	model__min_child_weight_1.5	0.0423
13	model__min_child_weight_2	0.0389
10	model__min_child_weight_0.75	0.0280
22	model__learning_rate_0.1	0.0260
26	model__learning_rate_0.5	0.0229
21	model__learning_rate_0.05	0.0222
9	model__min_child_weight_0.5	0.0208

Parameter hypertuning results:

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_model__scale_pos_weight	param_model__n_estimators	param_model__min_child_weight	param_model__max_depth	param_model__learning_rate	param_model__gamma	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
65	0.714997	0.020661	0.035049	0.006332	2	50	3.00	5	0.10	0.05	0.856604	0.833316	0.825936	0.812096	0.820978	0.829786	0.015080	1
17	6.111723	0.200491	0.095599	0.006802	3	150	3.00	10	0.05	0.10	0.842177	0.827341	0.814610	0.813087	0.807376	0.820918	0.012471	2
47	2.705106	0.046283	0.065339	0.003975	1	150	1.00	6	0.05	0.05	0.841391	0.814623	0.820488	0.810875	0.801109	0.817697	0.013419	3
46	6.393862	0.111464	0.087007	0.003882	2	100	0.75	12	0.10	0.10	0.833655	0.823808	0.813708	0.805664	0.801560	0.815679	0.011771	4
68	2.613279	0.204805	0.042049	0.004971	2	250	2.00	10	0.20	0.40	0.840833	0.818092	0.809786	0.809150	0.797954	0.815163	0.014344	5
1	9.582381	0.255538	0.108816	0.000892	2	200	2.50	10	0.01	0.40	0.835425	0.827289	0.805506	0.803940	0.802409	0.814914	0.013705	6
27	1.900886	0.131570	0.039376	0.006065	3	50	2.00	10	0.30	0.05	0.838977	0.828075	0.800602	0.801763	0.798999	0.813683	0.016587	7
52	3.585859	0.096698	0.082217	0.008712	2	200	1.00	6	0.05	0.40	0.832018	0.803565	0.817568	0.806282	0.801513	0.812189	0.011361	8
48	3.918704	0.124525	0.111475	0.018106	2	250	3.00	10	0.30	0.00	0.824241	0.817035	0.805680	0.814405	0.794350	0.811142	0.010284	9
64	3.881646	0.133103	0.100746	0.008933	1	250	1.50	12	0.40	0.00	0.834780	0.809736	0.801808	0.807396	0.799354	0.810615	0.012645	10

Because the hypertuning process is very slow (due to a very high number of fits required for transformer tuning and the size of the dataset) the tuning results are serialized in a binary file and used to train the test models separately.

2.1.1 Training and Testing the Default Risk Model

The dataset is pre-split into a 20% test sample and a 80% training sample (10% of which is used for tuning):

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

Total Samples Loaded 1210204

Training: XGBoostTuneF1 with: {'model__scale_pos_weight': 3, 'model__n_estimators': 50, 'model__min_child_weight': 2, 'model__max_depth': 10, 'model__learning_rate': 0.05, 'model__gamma': 0.05}
XGBoostF1Base: 98.2 seconds
Training: XGBoostTuneF1 with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 200, 'model__min_child_weight': 2, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostTuneF1: 79.0 seconds
Training: XGBoostTuneF1 with: {'feat_trans_dummy_drop_FICO__option': 1, 'model__scale_pos_weight': 5, 'model__n_estimators': 100, 'model__min_child_weight': 2, 'model__max_depth': 5, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostTuneF1DummyDropFICO: 92.5 seconds
Training: XGBoostTuneF1 with: {'model__scale_pos_weight': 10, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 12, 'model__learning_rate': 0.3, 'model__gamma': 0.3}
XGBoostPRAUCBase: 329.2 seconds
Training: XGBoostTunePRAUC with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0.05}
XGBoostTunePRAUC: 45.2 seconds

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:995: UserWarning: The figure layout has changed to tight
  plt.tight_layout()

Training: XGBoostTunePRAUC with: {'feat_trans_DropLowImportanceFeatures__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_purpose__option': 1, 'model__scale_pos_weight': 2, 'model__n_estimators': 50, 'model__min_child_weight': 3, 'model__max_depth': 5, 'model__learning_rate': 0.1, 'model__gamma': 0.05} XGBoostTunePRAUC: 1.1 seconds

The classification performance of all the models using the default 50% threshold is comparable (XGBoostTuneF1DummyDropFICO was included as a baseline).

We have chosen the XGBoostTunePRAUC configuration as our "production model" since it performs the best at predicting accurate probabilities (i.e. lowest log_loss).

XGBoostPRAUCBase (which achieves similar classification performance) uses a minimally processed dataset while XGBoostTunePRAUC uses several feature transformers (that were selected based on their impact on performances as part of our tuning process):

• DropLowImportanceFeatures, dropping features with importance close to 0.0 simplifies the model and has no negative impact on performance
• InstallementIncomeRatio, calculates the ratio between the income of the application and the expected installment size
• LoanPurposeRiskTransformer, uses target encoding for the Loan Purpose column by calculating the default risk based on it.

We have tried multiple other feature transformers (see src/feature_engineering.py) but they had negative or no signficant positive impact on performance.

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.081260
1	emp_length_parser__emp_length	0.006472
2	zip__zip_code	0.009154
3	pass__loan_amnt	0.004362
4	pass__int_rate	0.002490
5	pass__installment	0.005185
6	pass__grade	0.001119
7	pass__sub_grade	0.002440
8	pass__home_ownership	0.001933
9	pass__annual_inc	0.002389
10	pass__verification_status	0.002797
11	pass__purpose	0.001496
12	pass__addr_state	0.001898
13	pass__dti	0.002965
14	pass__delinq_2yrs	0.000711
15	pass__fico_range_low	0.003922
16	pass__fico_range_high	0.000000
17	pass__inq_last_6mths	0.001345
18	pass__mths_since_last_delinq	0.001392
19	pass__mths_since_last_record	0.001066
20	pass__open_acc	0.001103
21	pass__pub_rec	0.000344
22	pass__revol_bal	0.001249
23	pass__revol_util	0.001627
24	pass__total_acc	0.003171
25	pass__initial_list_status	0.003253
26	pass__last_fico_range_high	0.784888
27	pass__last_fico_range_low	0.000000
28	pass__collections_12_mths_ex_med	0.000303
29	pass__mths_since_last_major_derog	0.000990
30	pass__application_type	0.005523
31	pass__annual_inc_joint	0.004056
32	pass__dti_joint	0.002053
33	pass__verification_status_joint	0.007221
34	pass__inq_fi	0.010771
35	pass__inq_last_12m	0.029097
36	pass__chargeoff_within_12_mths	0.000204
37	pass__mort_acc	0.002018
38	pass__pub_rec_bankruptcies	0.002296
39	pass__tax_liens	0.000652
40	pass__tot_hi_cred_lim	0.002852
41	pass__total_bal_ex_mort	0.001932

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.041694
1	emp_length_parser__emp_length	0.005625
2	zip__zip_code	0.005618
3	pass__loan_amnt	0.033847
4	pass__int_rate	0.012316
5	pass__installment	0.021628
6	pass__grade	0.000000
7	pass__sub_grade	0.000000
8	pass__home_ownership	0.000000
9	pass__annual_inc	0.000000
10	pass__verification_status	0.000000
11	pass__purpose	0.000000
12	pass__addr_state	0.000000
13	pass__dti	0.020426
14	pass__delinq_2yrs	0.000000
15	pass__fico_range_low	0.012295
16	pass__fico_range_high	0.000000
17	pass__inq_last_6mths	0.000000
18	pass__mths_since_last_delinq	0.000000
19	pass__mths_since_last_record	0.000000
20	pass__open_acc	0.000000
21	pass__pub_rec	0.000000
22	pass__revol_bal	0.000000
23	pass__revol_util	0.000000
24	pass__total_acc	0.019025
25	pass__initial_list_status	0.000000
26	pass__last_fico_range_high	0.693645
27	pass__last_fico_range_low	0.000000
28	pass__collections_12_mths_ex_med	0.000000
29	pass__mths_since_last_major_derog	0.000000
30	pass__application_type	0.000000
31	pass__annual_inc_joint	0.007208
32	pass__dti_joint	0.000000
33	pass__verification_status_joint	0.005732
34	pass__inq_fi	0.039876
35	pass__inq_last_12m	0.059146
36	pass__chargeoff_within_12_mths	0.000000
37	pass__mort_acc	0.008540
38	pass__pub_rec_bankruptcies	0.000000
39	pass__tax_liens	0.000000
40	pass__tot_hi_cred_lim	0.010032
41	pass__total_bal_ex_mort	0.003348
42	purpose_risk_score	0.000000

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.031716
1	emp_length_parser__emp_length	0.008676
2	zip__zip_code	0.008679
3	pass__loan_amnt	0.012592
4	pass__int_rate	0.017748
5	pass__installment	0.012326
6	pass__grade	0.523029
7	pass__sub_grade	0.243649
8	pass__home_ownership	0.014115
9	pass__annual_inc	0.008138
10	pass__verification_status	0.000457
11	pass__purpose	0.000627
12	pass__addr_state	0.014832
13	pass__dti	0.013609
14	pass__delinq_2yrs	0.000000
15	pass__fico_range_low	0.007460
16	pass__fico_range_high	0.000000
17	pass__inq_last_6mths	0.000566
18	pass__mths_since_last_delinq	0.000679
19	pass__mths_since_last_record	0.000000
20	pass__open_acc	0.000615
21	pass__pub_rec	0.000000
22	pass__revol_bal	0.000000
23	pass__revol_util	0.000379
24	pass__total_acc	0.000572
25	pass__initial_list_status	0.000000
26	pass__collections_12_mths_ex_med	0.000000
27	pass__mths_since_last_major_derog	0.000166
28	pass__application_type	0.000000
29	pass__annual_inc_joint	0.000000
30	pass__dti_joint	0.000346
31	pass__verification_status_joint	0.000000
32	pass__inq_fi	0.025813
33	pass__inq_last_12m	0.016484
34	pass__chargeoff_within_12_mths	0.000000
35	pass__mort_acc	0.017178
36	pass__pub_rec_bankruptcies	0.000000
37	pass__tax_liens	0.000000
38	pass__tot_hi_cred_lim	0.018954
39	pass__total_bal_ex_mort	0.000596

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.147525
1	emp_length_parser__emp_length	0.010144
2	zip__zip_code	0.048551
3	pass__loan_amnt	0.009661
4	pass__int_rate	0.007145
5	pass__installment	0.009508
6	pass__grade	0.005471
7	pass__sub_grade	0.015495
8	pass__home_ownership	0.007210
9	pass__annual_inc	0.008128
10	pass__verification_status	0.007234
11	pass__purpose	0.009398
12	pass__addr_state	0.008266
13	pass__dti	0.007766
14	pass__delinq_2yrs	0.007853
15	pass__fico_range_low	0.009847
16	pass__fico_range_high	0.000000
17	pass__inq_last_6mths	0.007134
18	pass__mths_since_last_delinq	0.007088
19	pass__mths_since_last_record	0.008396
20	pass__open_acc	0.006907
21	pass__pub_rec	0.010293
22	pass__revol_bal	0.006926
23	pass__revol_util	0.007376
24	pass__total_acc	0.008453
25	pass__initial_list_status	0.007738
26	pass__last_fico_range_high	0.397840
27	pass__last_fico_range_low	0.000000
28	pass__collections_12_mths_ex_med	0.011604
29	pass__mths_since_last_major_derog	0.007780
30	pass__application_type	0.043336
31	pass__annual_inc_joint	0.028541
32	pass__dti_joint	0.018120
33	pass__verification_status_joint	0.013861
34	pass__inq_fi	0.014235
35	pass__inq_last_12m	0.019005
36	pass__chargeoff_within_12_mths	0.011032
37	pass__mort_acc	0.007319
38	pass__pub_rec_bankruptcies	0.011996
39	pass__tax_liens	0.010005
40	pass__tot_hi_cred_lim	0.008131
41	pass__total_bal_ex_mort	0.007683

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:457: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(

	Feature	Importance
0	term_parser__term	0.047196
1	emp_length_parser__emp_length	0.009523
2	zip__zip_code	0.007962
3	pass__loan_amnt	0.021693
4	pass__int_rate	0.011061
5	pass__installment	0.015877
6	pass__grade	0.007453
7	pass__sub_grade	0.002854
8	pass__home_ownership	0.005526
9	pass__annual_inc	0.010156
10	pass__verification_status	0.006244
11	pass__purpose	0.005823
12	pass__addr_state	0.001248
13	pass__dti	0.016517
14	pass__delinq_2yrs	0.000000
15	pass__fico_range_low	0.009993
16	pass__fico_range_high	0.000000
17	pass__inq_last_6mths	0.003550
18	pass__mths_since_last_delinq	0.002630
19	pass__mths_since_last_record	0.001133
20	pass__open_acc	0.000379
21	pass__pub_rec	0.000000
22	pass__revol_bal	0.003833
23	pass__revol_util	0.006724
24	pass__total_acc	0.015109
25	pass__initial_list_status	0.007440
26	pass__last_fico_range_high	0.643327
27	pass__last_fico_range_low	0.000000
28	pass__collections_12_mths_ex_med	0.000000
29	pass__mths_since_last_major_derog	0.000941
30	pass__application_type	0.000000
31	pass__annual_inc_joint	0.007815
32	pass__dti_joint	0.007615
33	pass__verification_status_joint	0.005253
34	pass__inq_fi	0.028570
35	pass__inq_last_12m	0.056947
36	pass__chargeoff_within_12_mths	0.000000
37	pass__mort_acc	0.005386
38	pass__pub_rec_bankruptcies	0.008609
39	pass__tax_liens	0.000000
40	pass__tot_hi_cred_lim	0.009844
41	pass__total_bal_ex_mort	0.003519
42	purpose_risk_score	0.002249

Analyzing and Predicting Grades

Average Default Rate by Grade in the Dataset:

	Counts	Proportion	Loan_Status_Ratio
A	211381	0.174666	0.060445
B	353495	0.292095	0.134030
C	343327	0.283693	0.224480
D	180840	0.149429	0.303639
E	84224	0.069595	0.384214
F	28719	0.023731	0.451165
G	8218	0.006791	0.501582

Evaluating Financial Returns/Performance

We'll use the model we have built to narrow down the sample of loans that have a lower risk to default.

This would be more useful to potential investors rather than the Lenders Club itself, they can use it in addition to the grade and other data provided by the Lenders Club to select the loans which not to invest it in.

The table below shows the total returns of all the loans in the dataset. We have to note that this is highly simplified because it assumes that all the loans that were defaulted on had to be completely written off but realistically this is not really the case:

• all lenders are likely to make at least some payments before they default
• defaulted loans can still be sold to third part for a fraction of the owed amount In future version of our model we should take these factors into account.

	Total Loan Amount	Total Interest Paid	Expected Return %	Total Return %	Total Loss
0	17451.88m	1770.28m	13.645651	10.143752	3757.82m

We have selected only the loans which are unlikely to default using the default 50% threshold (note that the predicted probabilities don't directly indicate the likelihood of default):

	Actual	Hypothetical
Total Loan Amount	17451.8813M	2590.8104M
Total Interest Paid	1770.2756M	330.8115M
Total Return %	10.14%	12.77%
Default Rate	19.96%	3.69%
Total Loss	3757.8181M	94.0324M
Losses Avoided	None	3663.7857M
Interest Lost	None	-1439.4641M
Total Applications Accepted	1210204	182757

Our model is fairly conservative and has rejected most of the loans offered by the Lenders Club (i.e. more than 80% are considered to be too risky). This would make it not very useful for the Lenders Club itself (however we might be able to improve this tuning the threshold). However, from the perspective of potential investors this is perfectly acceptable because they are likely to only invest into a small subset of available loans.

Using our model would allow investors to signficantly improve their returns by 3% or so, further improvements might be possible by improving the threshold (we'll look into this in the next section).

Calculating Probability Thresholds

If we're using our binary classification model to assign grades to loans based on default probabilities we can't directly try to predict the grades from the source dataset.

Basically what we can do is to use the source dataset as a baseline and try to determine whether our model can outperform the method the Lending Club is using for determine grades, we'll try to maintain the same ratio of grades while trying to minimize the default risk for loans included in top grades (basically we want to make sure that's the spread of default risk between grades is maximized i.e. default risk for grade A is as close to 0% and for grade F as close to 100% as possible while also ensuring that the risk increased with continuously with grade)

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1471: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))

The chart shows the performance of the if only individual with stroke Prob. > T are selected. Additionally the overlay indicates the number of people whose predicted P is in an given range. The overlays can be used to selected the most at risk individual based on the probability predicted for them

We have used the predicted probabilities to suggest alternative grades for each loan. Expected D.R is the default rate based on original grades. We can see our grades are much better at excluding higher risk loans from higher grades.

Generally this does not necessarily mean that source grades are inaccurate or objectively worse, the investors club is likely using a more straightforward and transparent algorithm for assigning them and investors should be using additional criteria when selecting which loans to invest anyway.

Comparing Interest Rates for Predicted Grades

In this section we'll try to evaluate investment strategies that are entirely based on loan grades (either the original ones or ones provided by our model)

Predicted Grade Real Return Rates

'Returns and Default Rate by Grade in Source Dataset:'

	default_rate	mean_interest_rate	expected_return
pass__grade_merged
A	0.06	7.11	6.68
B	0.14	10.69	9.23
C	0.22	14.03	10.87
D	0.30	17.73	12.37
E-F-G	0.41	22.47	13.35

'Returns and Default Rate Based on Default Probability Model:'

	default_rate	mean_interest_rate	expected_return
predicted_grades
A	0.003	10.357	10.326
B	0.009	12.271	12.164
C	0.082	13.943	12.800
D	0.559	14.714	6.495
E-F-G	0.901	16.862	1.672

We can see that while using our model to invested in the worst grades loans would lead to relatively poor returns investing in only in C grade or higher rated loans would provide better returns than using the Lenders Club's values to invest in the same loans:

• Investing only in C loans would provide almost the same returns as the far riskier E-F-G loans using the original grades.
• Only investing in A loans still yields ~10%, considering that the default rate is extremely low it might still be a preferable option to investing into high risk loans.

Basically using our model the investor would be able to minimize default risk while still getting decent returns i.e. if they only invested in loans with an estimated 10% default likelyhood or less using the original grades (so only A grade loans) they'd only make 6-7% while using our model this can be increased to more than 10%.

Limitations

We must take into account that this calculation is very simplified and therefore probably not accurate because, our model and calculations assume that Loss Given Default (LGD) is 100% and that Exposure at Default (EAD) is also 100% which is not accurate: - loans usually are not defaulted on immediately (i.e. some payments are almost always received) - most applicants who are unable to keep up with their payments are given some leeway and {PAYMENT PLAN} - failed loans are often sold to third parties for at least some fraction of the owed amount

Therefore, to provide a more accurate estimate and to figure out whether our model would actually be useful we'd need to incorporate the actual amounts received etc.

Total samples loaded : 1000

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

2    288
1    283
0    210
3    137
4     82
Name: target__grade, dtype: int64

Tunning: - transformers: transformers: 7
 total options: 15
[<class 'src.feature_engineering.LoanGradeTransformer'>, <class 'src.feature_engineering.JointApplicationTransformer'>, <class 'src.feature_engineering.FICOScoreTransformer'>, <class 'src.feature_engineering.DelinquencyTransformer'>, <class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.NewDtiTransformer'>, <class 'src.feature_engineering.DummyDropAllButFICOHigh'>]
search_grid:
{'feat_trans_loan_grade__option': [0, 1, 2], 'feat_trans_dti_inc_joint__option': [0, 1], 'feat_trans_fico_score__option': [0, 1], 'feat_trans_delinquency__option': [0, 1], 'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_new_dti_after_loan__option': [0, 1], 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': [0, 1]}

 - model: XGBoostOrdinalGradeRegressor_Default n_iters=50 with:
 {'model__learning_rate': [0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12, None], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4]}
Fitting 5 folds for each of 192 candidates, totalling 960 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
720 fits failed out of a total of 960.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 81, in transform
    df[col] = _calc_loan_grade(df[col], False)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 83, in transform
    df[col] = _calc_loan_grade(df[col], True)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
80 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 274, in transform
    df = df[["pass__last_fico_range_high"]].copy()
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['pass__last_fico_range_high'], dtype='object')] are in the [columns]"

  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [-0.88042996 -0.87518446         nan         nan         nan         nan
 -0.84446263 -0.85909352         nan         nan         nan         nan
 -0.88042996 -0.87518446         nan         nan         nan         nan
 -0.84446263 -0.85909352         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -0.89201032 -0.85979639         nan         nan         nan         nan
 -0.86618961 -0.84521511         nan         nan         nan         nan
 -0.89201032 -0.85979639         nan         nan         nan         nan
 -0.86618961 -0.84521511         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -0.88678709 -0.88353414         nan         nan         nan         nan
 -0.8461568  -0.85376689         nan         nan         nan         nan
 -0.88678709 -0.88353414         nan         nan         nan         nan
 -0.8461568  -0.85376689         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -0.90117584 -0.86268431         nan         nan         nan         nan
 -0.86789756 -0.84720845         nan         nan         nan         nan
 -0.90117584 -0.86268431         nan         nan         nan         nan
 -0.86789756 -0.84720845         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
 -1.24844423 -1.24844423         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0}
Fitting 5 folds for each of 50 candidates, totalling 250 fits
Best Model Parameters: {'model__n_estimators': 150, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostOrdinalGradeRegressor_Default Fit, total time:36.8

Tunning: - transformers: transformers: 7
 total options: 15
[<class 'src.feature_engineering.LoanGradeTransformer'>, <class 'src.feature_engineering.JointApplicationTransformer'>, <class 'src.feature_engineering.FICOScoreTransformer'>, <class 'src.feature_engineering.DelinquencyTransformer'>, <class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.NewDtiTransformer'>, <class 'src.feature_engineering.DummyDropAllButFICOHigh'>]
search_grid:
{'feat_trans_loan_grade__option': [0, 1, 2], 'feat_trans_dti_inc_joint__option': [0, 1], 'feat_trans_fico_score__option': [0, 1], 'feat_trans_delinquency__option': [0, 1], 'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_new_dti_after_loan__option': [0, 1], 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': [0, 1]}

 - model: XGBoostMulticlassTunePRAUC n_iters=50 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4]}
Fitting 5 folds for each of 192 candidates, totalling 960 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
720 fits failed out of a total of 960.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 81, in transform
    df[col] = _calc_loan_grade(df[col], False)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 83, in transform
    df[col] = _calc_loan_grade(df[col], True)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
80 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 274, in transform
    df = df[["pass__last_fico_range_high"]].copy()
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['pass__last_fico_range_high'], dtype='object')] are in the [columns]"

  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [0.39865792 0.38915238        nan        nan        nan        nan
 0.39865792 0.38915238        nan        nan        nan        nan
 0.39865792 0.38915238        nan        nan        nan        nan
 0.39865792 0.38915238        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
 0.3892805  0.37535732        nan        nan        nan        nan
 0.3901784  0.38983167        nan        nan        nan        nan
 0.3892805  0.37535732        nan        nan        nan        nan
 0.3901784  0.38983167        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
 0.39174443 0.39191987        nan        nan        nan        nan
 0.39174443 0.39191987        nan        nan        nan        nan
 0.39174443 0.39191987        nan        nan        nan        nan
 0.39174443 0.39191987        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
 0.40028704 0.38580882        nan        nan        nan        nan
 0.40028689 0.38949293        nan        nan        nan        nan
 0.40028704 0.38580882        nan        nan        nan        nan
 0.40028689 0.38949293        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
 0.24410438 0.24410438        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan
        nan        nan        nan        nan        nan        nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0}
Fitting 5 folds for each of 50 candidates, totalling 250 fits
Best Model Parameters: {'model__n_estimators': 150, 'model__min_child_weight': 0.5, 'model__max_depth': 12, 'model__learning_rate': 0.2, 'model__gamma': 0}
XGBoostMulticlassTunePRAUC Fit, total time:101.2

Tunning: - transformers: transformers: 7
 total options: 15
[<class 'src.feature_engineering.LoanGradeTransformer'>, <class 'src.feature_engineering.JointApplicationTransformer'>, <class 'src.feature_engineering.FICOScoreTransformer'>, <class 'src.feature_engineering.DelinquencyTransformer'>, <class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.NewDtiTransformer'>, <class 'src.feature_engineering.DummyDropAllButFICOHigh'>]
search_grid:
{'feat_trans_loan_grade__option': [0, 1, 2], 'feat_trans_dti_inc_joint__option': [0, 1], 'feat_trans_fico_score__option': [0, 1], 'feat_trans_delinquency__option': [0, 1], 'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_new_dti_after_loan__option': [0, 1], 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': [0, 1]}

 - model: XGBoostMulticlassTuneLogLoss n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4]}
Fitting 5 folds for each of 192 candidates, totalling 960 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
720 fits failed out of a total of 960.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 81, in transform
    df[col] = _calc_loan_grade(df[col], False)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 83, in transform
    df[col] = _calc_loan_grade(df[col], True)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
80 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 274, in transform
    df = df[["pass__last_fico_range_high"]].copy()
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['pass__last_fico_range_high'], dtype='object')] are in the [columns]"

  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [-1.33395429 -1.33670064         nan         nan         nan         nan
 -1.33395429 -1.33670064         nan         nan         nan         nan
 -1.33395429 -1.33670064         nan         nan         nan         nan
 -1.33395429 -1.33670064         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -1.33603874 -1.3392099          nan         nan         nan         nan
 -1.33488397 -1.34099195         nan         nan         nan         nan
 -1.33603874 -1.3392099          nan         nan         nan         nan
 -1.33488397 -1.34099195         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -1.33409228 -1.34237142         nan         nan         nan         nan
 -1.33409228 -1.34237142         nan         nan         nan         nan
 -1.33409228 -1.34237142         nan         nan         nan         nan
 -1.33409228 -1.34237142         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
 -1.33462439 -1.33826324         nan         nan         nan         nan
 -1.33244639 -1.33767518         nan         nan         nan         nan
 -1.33462439 -1.33826324         nan         nan         nan         nan
 -1.33244639 -1.33767518         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
 -1.48575592 -1.48575592         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan
         nan         nan         nan         nan         nan         nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0}
Fitting 5 folds for each of 70 candidates, totalling 350 fits
Best Model Parameters: {'model__n_estimators': 100, 'model__min_child_weight': 2.5, 'model__max_depth': 4, 'model__learning_rate': 0.05, 'model__gamma': 0}
XGBoostMulticlassTuneLogLoss Fit, total time:118.8

Tunning: - transformers: transformers: 7
 total options: 15
[<class 'src.feature_engineering.LoanGradeTransformer'>, <class 'src.feature_engineering.JointApplicationTransformer'>, <class 'src.feature_engineering.FICOScoreTransformer'>, <class 'src.feature_engineering.DelinquencyTransformer'>, <class 'src.feature_engineering.InstallementIncomeRatio'>, <class 'src.feature_engineering.NewDtiTransformer'>, <class 'src.feature_engineering.DummyDropAllButFICOHigh'>]
search_grid:
{'feat_trans_loan_grade__option': [0, 1, 2], 'feat_trans_dti_inc_joint__option': [0, 1], 'feat_trans_fico_score__option': [0, 1], 'feat_trans_delinquency__option': [0, 1], 'feat_trans_inst_income_ratio__option': [0, 1], 'feat_trans_new_dti_after_loan__option': [0, 1], 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': [0, 1]}

 - model: XGBoostF1Multiclass n_iters=70 with:
 {'model__learning_rate': [0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 1], 'model__max_depth': [4, 5, 6, 7, 10, 12], 'model__n_estimators': [50, 100, 150, 200, 250], 'model__min_child_weight': [0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3], 'model__gamma': [0, 0.05, 0.1, 0.3, 0.4]}
Fitting 5 folds for each of 192 candidates, totalling 960 fits

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:425: FitFailedWarning: 
720 fits failed out of a total of 960.
The score on these train-test partitions for these parameters will be set to nan.
If these failures are not expected, you can try to debug them by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 81, in transform
    df[col] = _calc_loan_grade(df[col], False)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
320 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pass__grade'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 83, in transform
    df[col] = _calc_loan_grade(df[col], True)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'pass__grade'

--------------------------------------------------------------------------------
80 fits failed with the following error:
Traceback (most recent call last):
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 729, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 423, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 377, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/joblib/memory.py", line 353, in __call__
    return self.func(*args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/base.py", line 919, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/home/paulius/data/projects/m3_s3_lending/src/feature_engineering.py", line 274, in transform
    df = df[["pass__last_fico_range_high"]].copy()
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['pass__last_fico_range_high'], dtype='object')] are in the [columns]"

  warnings.warn(some_fits_failed_message, FitFailedWarning)
/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/model_selection/_search.py:979: UserWarning: One or more of the test scores are non-finite: [0.442 0.441   nan   nan   nan   nan 0.442 0.441   nan   nan   nan   nan
 0.442 0.441   nan   nan   nan   nan 0.442 0.441   nan   nan   nan   nan
 0.33  0.33    nan   nan   nan   nan 0.33  0.33    nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
 0.431 0.421   nan   nan   nan   nan 0.427 0.429   nan   nan   nan   nan
 0.431 0.421   nan   nan   nan   nan 0.427 0.429   nan   nan   nan   nan
 0.33  0.33    nan   nan   nan   nan 0.33  0.33    nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
 0.437 0.436   nan   nan   nan   nan 0.437 0.436   nan   nan   nan   nan
 0.437 0.436   nan   nan   nan   nan 0.437 0.436   nan   nan   nan   nan
 0.33  0.33    nan   nan   nan   nan 0.33  0.33    nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan
 0.435 0.422   nan   nan   nan   nan 0.442 0.425   nan   nan   nan   nan
 0.435 0.422   nan   nan   nan   nan 0.442 0.425   nan   nan   nan   nan
 0.33  0.33    nan   nan   nan   nan 0.33  0.33    nan   nan   nan   nan
   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan   nan]
  warnings.warn(

Best Transformer Parameters: {'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0}
Fitting 5 folds for each of 70 candidates, totalling 350 fits
Best Model Parameters: {'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}
XGBoostF1Multiclass Fit, total time:302.4

	best_score	best_params
model_key
XGBoostOrdinalGradeRegressor_Default	-1.304155	{'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__max_depth': 4, 'model__learning_rate': 0.01, 'model__gamma': 0}
XGBoostMulticlassTunePRAUC	0.407976	{'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 0.5, 'model__max_depth': 12, 'model__learning_rate': 0.2, 'model__gamma': 0}
XGBoostMulticlassTuneLogLoss	-1.287763	{'feat_trans_delinquency__option': 1, 'feat_trans_dti_inc_joint__option': 1, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 1, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 100, 'model__min_child_weight': 2.5, 'model__max_depth': 4, 'model__learning_rate': 0.05, 'model__gamma': 0}
XGBoostF1Multiclass	0.439000	{'feat_trans_delinquency__option': 0, 'feat_trans_dti_inc_joint__option': 0, 'feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option': 0, 'feat_trans_fico_score__option': 0, 'feat_trans_inst_income_ratio__option': 0, 'feat_trans_loan_grade__option': 0, 'feat_trans_new_dti_after_loan__option': 0, 'model__n_estimators': 150, 'model__min_child_weight': 2.5, 'model__max_depth': 7, 'model__learning_rate': 0.3, 'model__gamma': 0.1}

'\n Model: XGBoostOrdinalGradeRegressor_Default'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	-1.248444
169	1	1	1	0	0	0	1	-1.248444
168	1	1	1	0	0	0	0	-1.248444
127	1	0	1	0	1	0	1	-1.248444
126	1	0	1	0	1	0	0	-1.248444
121	1	0	1	0	0	0	1	-1.248444
120	1	0	1	0	0	0	0	-1.248444
174	1	1	1	0	1	0	0	-1.248444
78	0	1	1	0	1	0	0	-1.248444
73	0	1	1	0	0	0	1	-1.248444

'Transformer impact for XGBoostOrdinalGradeRegressor_Default'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0176
5	feat_trans_new_dti_after_loan__option_1	0.0082
3	feat_trans_fico_score__option_1	0.0000
1	feat_trans_dti_inc_joint__option_1	-0.0011
0	feat_trans_delinquency__option_1	-0.0022
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.3815

XGBoostOrdinalGradeRegressor_Default

'\n Model: XGBoostMulticlassTunePRAUC'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	0.244104
169	1	1	1	0	0	0	1	0.244104
168	1	1	1	0	0	0	0	0.244104
127	1	0	1	0	1	0	1	0.244104
126	1	0	1	0	1	0	0	0.244104
121	1	0	1	0	0	0	1	0.244104
120	1	0	1	0	0	0	0	0.244104
174	1	1	1	0	1	0	0	0.244104
78	0	1	1	0	1	0	0	0.244104
73	0	1	1	0	0	0	1	0.244104

'Transformer impact for XGBoostMulticlassTunePRAUC'

	Transformer_Option	Impact
0	feat_trans_delinquency__option_1	0.0019
4	feat_trans_inst_income_ratio__option_1	0.0016
3	feat_trans_fico_score__option_1	0.0000
1	feat_trans_dti_inc_joint__option_1	-0.0019
5	feat_trans_new_dti_after_loan__option_1	-0.0049
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1474

XGBoostMulticlassTunePRAUC

'\n Model: XGBoostMulticlassTuneLogLoss'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	-1.485756
169	1	1	1	0	0	0	1	-1.485756
168	1	1	1	0	0	0	0	-1.485756
127	1	0	1	0	1	0	1	-1.485756
126	1	0	1	0	1	0	0	-1.485756
121	1	0	1	0	0	0	1	-1.485756
120	1	0	1	0	0	0	0	-1.485756
174	1	1	1	0	1	0	0	-1.485756
78	0	1	1	0	1	0	0	-1.485756
73	0	1	1	0	0	0	1	-1.485756

'Transformer impact for XGBoostMulticlassTuneLogLoss'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0002
1	feat_trans_dti_inc_joint__option_1	0.0000
3	feat_trans_fico_score__option_1	0.0000
0	feat_trans_delinquency__option_1	-0.0003
5	feat_trans_new_dti_after_loan__option_1	-0.0033
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1490

XGBoostMulticlassTuneLogLoss

'\n Model: XGBoostF1Multiclass'

	feat_trans_delinquency__option	feat_trans_dti_inc_joint__option	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option	feat_trans_fico_score__option	feat_trans_inst_income_ratio__option	feat_trans_loan_grade__option	feat_trans_new_dti_after_loan__option	mean_test_score
79	0	1	1	0	1	0	1	0.33
169	1	1	1	0	0	0	1	0.33
168	1	1	1	0	0	0	0	0.33
127	1	0	1	0	1	0	1	0.33
126	1	0	1	0	1	0	0	0.33
121	1	0	1	0	0	0	1	0.33
120	1	0	1	0	0	0	0	0.33
174	1	1	1	0	1	0	0	0.33
78	0	1	1	0	1	0	0	0.33
73	0	1	1	0	0	0	1	0.33

'Transformer impact for XGBoostF1Multiclass'

	Transformer_Option	Impact
4	feat_trans_inst_income_ratio__option_1	0.0012
3	feat_trans_fico_score__option_1	0.0000
0	feat_trans_delinquency__option_1	-0.0003
5	feat_trans_new_dti_after_loan__option_1	-0.0035
1	feat_trans_dti_inc_joint__option_1	-0.0067
2	feat_trans_dummy_DROP_ALL_BUT_FICO_HIGH__option_1	-0.1040

XGBoostF1Multiclass

1.1 Features

This is a supplement to the main EDA notebook (1.2 EDA) that includes the individual analysis of all numerical and categorical variables. The insights obtained here together with the analysis of interaction between individual variables was used to build the preprocessing pipeline and individual transformers (for feature engineering) that were used when building our models (see the EDA notebook for additional details).

Dropping transformer for ['term'], not in df

/home/paulius/miniconda3/envs/rapids_v2/lib/python3.10/site-packages/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(

	emp_length_parser__emp_length	target__Accepted	zip__zip_code	pass__loan_amnt	pass__addr_state	pass__last_fico_range_high	pass__dti
133128	NaN	1	863	10000.0	AZ	824.0	0.2640
734065	2.0	1	208	6500.0	MD	819.0	0.0730
714506	11.0	1	275	8400.0	NC	709.0	0.1729
1866411	11.0	1	750	31000.0	TX	714.0	0.2701
3391910	0.0	0	769	7500.0	TX	656.0	0.4015

'Features with NaN Values:'

	Total NaN Values	Proportion NaN (%)
emp_length_parser__emp_length	457	5.0
pass__last_fico_range_high	90	1.0
pass__dti	7	0.0

'Total Duplicate Rows: 0.06'

	emp_length_parser__emp_length	target__Accepted	zip__zip_code	pass__loan_amnt	pass__addr_state	pass__last_fico_range_high	pass__dti
2654687	0.0	0	286	5000.0	NC	0.0	0.00
2907914	0.0	0	100	35000.0	NY	NaN	-0.01
3087120	0.0	0	331	1000.0	FL	0.0	0.00
3043169	0.0	0	100	35000.0	NY	NaN	-0.01
2724606	0.0	0	331	1000.0	FL	0.0	0.00
3135218	0.0	0	286	5000.0	NC	0.0	0.00

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:1046: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  axes[0].set_xticklabels(tick_labels)

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:1036: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  count_plot_df[variable] = count_plot_df[variable].astype("str")

/home/paulius/data/projects/m3_s3_lending/shared/graph.py:1036: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  count_plot_df[variable] = count_plot_df[variable].astype("str")

1.1 Features

This is a supplement to the main EDA notebook (1.2 EDA) that includes the individual analysis of all numerical and categorical variables. The insights obtained here together with the analysis of interaction between individual variables was used to build the preprocessing pipeline and individual transformers (for feature engineering) that were used when building our models (see the EDA notebook for additional details).

With transform="pandas", `func` should return a DataFrame to follow the set_output API.

	term_parser__term	emp_length_parser__emp_length	target__grade	pass__loan_amnt	pass__int_rate	pass__installment	pass__sub_grade	pass__home_ownership	pass__annual_inc	pass__verification_status	pass__purpose	pass__zip_code	pass__addr_state	pass__dti	pass__delinq_2yrs	pass__fico_range_low	pass__fico_range_high	pass__inq_last_6mths	pass__mths_since_last_delinq	pass__mths_since_last_record	pass__open_acc	pass__pub_rec	pass__revol_bal	pass__revol_util	pass__total_acc	pass__initial_list_status	pass__last_fico_range_high	pass__last_fico_range_low	pass__collections_12_mths_ex_med	pass__mths_since_last_major_derog	pass__application_type	pass__annual_inc_joint	pass__dti_joint	pass__verification_status_joint	pass__inq_fi	pass__inq_last_12m	pass__chargeoff_within_12_mths	pass__mort_acc	pass__pub_rec_bankruptcies	pass__tax_liens	pass__tot_hi_cred_lim	pass__total_bal_ex_mort	pass__loan_status
2086299	36	1.0	0	14000.0	6.08	426.42	A2	MORTGAGE	70000.0	Not Verified	debt_consolidation	176xx	PA	31.10	0.0	800.0	804.0	0.0	NaN	NaN	18.0	0.0	22237.0	32.3	33.0	w	824.0	820.0	0.0	NaN	Joint App	350000.0	8.61	Not Verified	1.0	0.0	0.0	1.0	0.0	0.0	492126.0	78093.0	Current
1233152	60	NaN	4	35000.0	19.52	917.97	E2	MORTGAGE	120000.0	Verified	debt_consolidation	799xx	TX	23.75	1.0	710.0	714.0	3.0	22.0	NaN	11.0	0.0	49315.0	43.7	34.0	f	804.0	800.0	0.0	NaN	Individual	NaN	NaN	None	NaN	NaN	0.0	3.0	0.0	0.0	390414.0	90585.0	Fully Paid
461729	36	10.0	0	10000.0	7.34	310.33	A4	MORTGAGE	70000.0	Not Verified	debt_consolidation	063xx	CT	14.72	0.0	685.0	689.0	0.0	NaN	106.0	6.0	1.0	21943.0	47.5	16.0	w	749.0	745.0	0.0	NaN	Individual	NaN	NaN	None	0.0	0.0	0.0	1.0	1.0	0.0	56200.0	26022.0	Fully Paid
738295	60	10.0	3	21600.0	17.27	539.96	D2	MORTGAGE	96000.0	Verified	debt_consolidation	482xx	MI	15.83	1.0	690.0	694.0	1.0	15.0	NaN	11.0	0.0	14751.0	71.3	32.0	w	639.0	635.0	0.0	42.0	Individual	NaN	NaN	None	3.0	1.0	0.0	3.0	0.0	0.0	242183.0	76547.0	Fully Paid
864585	36	3.0	1	3300.0	10.07	106.60	B2	RENT	22000.0	Verified	other	490xx	MI	2.73	0.0	730.0	734.0	0.0	NaN	NaN	2.0	0.0	680.0	42.5	3.0	w	764.0	760.0	0.0	NaN	Individual	NaN	NaN	None	0.0	0.0	0.0	0.0	0.0	0.0	1600.0	680.0	Current

'Features with NaN Values:'

	Total NaN Values	Proportion NaN (%)
emp_length_parser__emp_length	1325	15.0
pass__dti	2	0.0
pass__mths_since_last_delinq	4587	51.0
pass__mths_since_last_record	7593	84.0
pass__revol_util	4	0.0
pass__mths_since_last_major_derog	6671	74.0
pass__annual_inc_joint	8545	95.0
pass__dti_joint	8545	95.0
pass__verification_status_joint	8556	95.0
pass__inq_fi	3422	38.0
pass__inq_last_12m	3422	38.0
pass__mort_acc	179	2.0
pass__pub_rec_bankruptcies	7	0.0
pass__tot_hi_cred_lim	275	3.0
pass__total_bal_ex_mort	179	2.0

	Total NaN Values	Proportion NaN (%)
pass__verification_status_joint	8556	95.0

count              444
unique               3
top       Not Verified
freq               212
Name: pass__verification_status_joint, dtype: object

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.